In verifying a client/server system, such as an NMS/EMS system, isolation of a discovered problem may be difficult. It may be far from clear whether the problem arose in the client software, the server software, the interface between them, or some combination of client, server, and interface. This is particularly true in complex systems such as Alcatel 5620™ (NMS) and Alcatel AWS™ (EMS) systems. Most of the feature testing and automated testing of the 5620/AWS system are at the system level. The system should have all the features incorporated before the automated feature level testing can be used.
The interface between the NMS and EMS is well defined. When new features are added, the interface is updated as the NMS development team specifies new interactions necessary between the NMS and the EMS to support the feature. However, at the end of the development cycle and during feature testing, if a complex behavior fails it becomes difficult to understand whether the problem arose due to bad logic on the client side or on the server side, or whether the problem was due to a bad interface specification.
One common method of automated testing is to record a set of interactions with the system and then use the recorded data to repeat the test again and verify the responses based on the recorded data. Although simple, this approach has disadvantages. If the system behavior changes, such as by adding a new feature, there is no recorded data with which to compare new test results. Only once a feature is fully implemented can data be captured for a scenario for use in testing the scenario. This also requires a knowledgeable individual to manually verify the captured data for the new feature.
Another method is functional decomposition. Key functional aspects of the system are identified and scripts are written to activate those areas. A test harness is written which performs a set of actions by invoking APIs of the interface. The test harness must encode all the data values that a function can return and compare the response received against the encoded data set. However, creating such a test harness requires a significant amount of work, and the test harness is also tightly coupled with the environment in which the EMS system will be operated. For example, if CORBA is used as the interface, the EMS and the test harness must use the same ORB provider, and if the EMS uses any special customization or non-standard feature of a particular ORB then these issues need to be resolved before the test harness can be executed.
A simple method of testing the interface directly using only the client or the server in isolation would allow problems with the interface to be detected early in the system development cycle. Such a method should also only require loose coupling between the test environment and the EMS, such as by not linking with the EMS environment at run time. Once the interface is known to be working correctly, system level testing can focus on the logic within the client or server. This in turn would shorten the software development cycle.